Tradition systems for testing network-connected devices, especially resource-limited mobile devices, typically utilize a test framework that may include a set of test harness plug-ins providing support for the platform or operation system under test. A testing framework may use the test harness for test execution and test suite management.
Frequently, the total number of tests in a test suite is extremely large, such as might be due to the variety and complexity of the platform and/or implementation being tested. Generally, tests may be executed by downloading a test application and running it on the test device. The total time required to execute a test suite may depend on the number of test applications being downloaded because the overall execution time may increase with increased network connectivity, as well the time needed for installation and/or launching of each test application. This may be especially significant for test devices with limited processing power, memory, and network bandwidth.
Traditionally, test frameworks require that the data for each test and the results of executing that test be communicated separately, from each other and from test data and results for other tests. That is, the test data for a particular test is communicated using a data channel established specifically for communicating that test's test data. Similarly, a new data channel may be established for communicating the data results of a particular test. Thus, even if multiple test applications may be bundled together for transfer, each test being executed may require two additional connections—one to communicate test data and another to communicate test results after test execution.
In some cases the total number of connections may equal the number of application downloads plus twice the number of tests in the test suite. For example, a test suite with 10000 tests and 50 tests per a bundle may require at least 200 connections (e.g., 10000 tests/50 tests per bundle) for downloading the test applications plus 20000 connections for test execution (e.g., 2 connections per each 10000 test). The large number of required connections may significantly increase the test suite execution time, especially for testing devices that have limited processing power, memory, and network bandwidth.